{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Serving a TensorFlow Model as a REST Endpoint with TensorFlow Serving and SageMaker\n",
    "\n",
    "We need to understand the application and business context to choose between real-time and batch predictions. Are we trying to optimize for latency or throughput? Does the application require our models to scale automatically throughout the day to handle cyclic traffic requirements? Do we plan to compare models in production through A/B tests?\n",
    "\n",
    "If our application requires low latency, then we should deploy the model as a real-time API to provide super-fast predictions on single prediction requests over HTTPS. We can deploy, scale, and compare our model prediction servers with SageMaker Endpoints."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/sagemaker-architecture.png\" width=\"80%\" align=\"left\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q sagemaker==2.9.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "import pandas as pd\n",
    "\n",
    "sess   = sagemaker.Session()\n",
    "bucket = sess.default_bucket()\n",
    "role = sagemaker.get_execution_role()\n",
    "region = boto3.Session().region_name\n",
    "\n",
    "sm = boto3.Session().client(service_name='sagemaker', region_name=region)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r recommender_training_job_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[OK]\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    recommender_training_job_name\n",
    "    print('[OK]')\n",
    "except NameError:\n",
    "    print('+++++++++++++++++++++++++++++++')\n",
    "    print('[ERROR] Please run the notebooks in the previous TRAIN section before you continue.')\n",
    "    print('+++++++++++++++++++++++++++++++')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensorflow-training-201129-2302-005-aec2a92a\n"
     ]
    }
   ],
   "source": [
    "print(recommender_training_job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Copy the Model from S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "download: s3://sagemaker-us-east-1-835319576252/tensorflow-training-201129-2302-005-aec2a92a/output/model.tar.gz to ./model.tar.gz\n"
     ]
    }
   ],
   "source": [
    "!aws s3 cp s3://$bucket/$recommender_training_job_name/output/model.tar.gz ./model.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensorflow/\n",
      "tensorflow/saved_model/\n",
      "tensorflow/saved_model/0/\n",
      "tensorflow/saved_model/0/assets/\n",
      "tensorflow/saved_model/0/variables/\n",
      "tensorflow/saved_model/0/variables/variables.index\n",
      "tensorflow/saved_model/0/variables/variables.data-00000-of-00001\n",
      "tensorflow/saved_model/0/saved_model.pb\n",
      "tensorboard/\n",
      "tensorboard/train/\n",
      "tensorboard/train/plugins/\n",
      "tensorboard/train/plugins/profile/\n",
      "tensorboard/train/plugins/profile/2020_11_29_23_19_30/\n",
      "tensorboard/train/plugins/profile/2020_11_29_23_19_30/ip-10-2-212-34.ec2.internal.tensorflow_stats.pb\n",
      "tensorboard/train/plugins/profile/2020_11_29_23_19_30/ip-10-2-212-34.ec2.internal.input_pipeline.pb\n",
      "tensorboard/train/plugins/profile/2020_11_29_23_19_30/ip-10-2-212-34.ec2.internal.overview_page.pb\n",
      "tensorboard/train/plugins/profile/2020_11_29_23_19_30/ip-10-2-212-34.ec2.internal.kernel_stats.pb\n",
      "tensorboard/train/plugins/profile/2020_11_29_23_19_30/ip-10-2-212-34.ec2.internal.memory_profile.json.gz\n",
      "tensorboard/train/plugins/profile/2020_11_29_23_19_30/ip-10-2-212-34.ec2.internal.xplane.pb\n",
      "tensorboard/train/plugins/profile/2020_11_29_23_19_30/ip-10-2-212-34.ec2.internal.trace.json.gz\n",
      "tensorboard/train/events.out.tfevents.1606691970.ip-10-2-212-34.ec2.internal.profile-empty\n",
      "tensorboard/train/events.out.tfevents.1606691968.ip-10-2-212-34.ec2.internal.33.452.v2\n",
      "code/\n",
      "code/inference.py\n"
     ]
    }
   ],
   "source": [
    "!mkdir -p ./deployed_model/\n",
    "!tar -xvzf ./model.tar.gz -C ./deployed_model/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-30 00:01:51.224430: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory\n",
      "2020-11-30 00:01:51.224474: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "\n",
      "MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:\n",
      "\n",
      "signature_def['__saved_model_init_op']:\n",
      "  The given SavedModel SignatureDef contains the following input(s):\n",
      "  The given SavedModel SignatureDef contains the following output(s):\n",
      "    outputs['__saved_model_init_op'] tensor_info:\n",
      "        dtype: DT_INVALID\n",
      "        shape: unknown_rank\n",
      "        name: NoOp\n",
      "  Method name is: \n",
      "\n",
      "signature_def['serving_default']:\n",
      "  The given SavedModel SignatureDef contains the following input(s):\n",
      "    inputs['input_1'] tensor_info:\n",
      "        dtype: DT_STRING\n",
      "        shape: (-1)\n",
      "        name: serving_default_input_1:0\n",
      "  The given SavedModel SignatureDef contains the following output(s):\n",
      "    outputs['output_1'] tensor_info:\n",
      "        dtype: DT_FLOAT\n",
      "        shape: (-1, 10)\n",
      "        name: StatefulPartitionedCall:0\n",
      "    outputs['output_2'] tensor_info:\n",
      "        dtype: DT_STRING\n",
      "        shape: (-1, 10)\n",
      "        name: StatefulPartitionedCall:1\n",
      "  Method name is: tensorflow/serving/predict\n",
      "2020-11-30 00:01:55.172037: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory\n",
      "2020-11-30 00:01:55.172091: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "2020-11-30 00:01:55.172132: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (datascience-1-0-ml-t3-medium-1abf3407f667f989be9d86559395): /proc/driver/nvidia/version does not exist\n",
      "Traceback (most recent call last):\n",
      "  File \"/opt/conda/bin/saved_model_cli\", line 8, in <module>\n",
      "    sys.exit(main())\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/tensorflow/python/tools/saved_model_cli.py\", line 1185, in main\n",
      "    args.func(args)\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/tensorflow/python/tools/saved_model_cli.py\", line 715, in show\n",
      "    _show_all(args.dir)\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/tensorflow/python/tools/saved_model_cli.py\", line 307, in _show_all\n",
      "    _show_defined_functions(saved_model_dir)\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/tensorflow/python/tools/saved_model_cli.py\", line 187, in _show_defined_functions\n",
      "    trackable_object = load.load(saved_model_dir)\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py\", line 603, in load\n",
      "    return load_internal(export_dir, tags, options)\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py\", line 633, in load_internal\n",
      "    ckpt_options)\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py\", line 131, in __init__\n",
      "    self._restore_checkpoint()\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py\", line 358, in _restore_checkpoint\n",
      "    \"%r from the checkpoint.\" % obj))\n",
      "NotImplementedError: Missing functionality to restore state of object <tensorflow.python.saved_model.load._RestoredResource object at 0x7fc0d2a26690> from the checkpoint.\n"
     ]
    }
   ],
   "source": [
    "!saved_model_cli show --all --dir ./deployed_model/tensorflow/saved_model/0/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_id = \"42\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-30 00:01:57.402673: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory\n",
      "2020-11-30 00:01:57.402719: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "2020-11-30 00:02:00.084853: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory\n",
      "2020-11-30 00:02:00.084909: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "2020-11-30 00:02:00.084952: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (datascience-1-0-ml-t3-medium-1abf3407f667f989be9d86559395): /proc/driver/nvidia/version does not exist\n",
      "2020-11-30 00:02:00.085328: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2020-11-30 00:02:00.122508: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2499995000 Hz\n",
      "2020-11-30 00:02:00.122752: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5620701cd400 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n",
      "2020-11-30 00:02:00.122863: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n",
      "WARNING:tensorflow:From /opt/conda/lib/python3.7/site-packages/tensorflow/python/tools/saved_model_cli.py:444: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.\n",
      "INFO:tensorflow:Restoring parameters from ./deployed_model/tensorflow/saved_model/0/variables/variables\n",
      "Result for output key output_1:\n",
      "[[0.02245261 0.02220378 0.01830344 0.01793418 0.01775248 0.01731658\n",
      "  0.01677447 0.01676866 0.01675068 0.01655273]]\n",
      "Result for output key output_2:\n",
      "[[b'Half Baked (1998)' b'Underneath, The (1995)' b'Baton Rouge (1988)'\n",
      "  b'Leading Man, The (1996)' b'Firm, The (1993)'\n",
      "  b'Sex, Lies, and Videotape (1989)' b'Ransom (1996)'\n",
      "  b\"Wooden Man's Bride, The (Wu Kui) (1994)\" b'Cable Guy, The (1996)'\n",
      "  b'Four Weddings and a Funeral (1994)']]\n"
     ]
    }
   ],
   "source": [
    "!saved_model_cli run --input_exprs 'input_1=np.array([\"$user_id\"])' --tag_set serve --signature_def serving_default --dir ./deployed_model/tensorflow/saved_model/0/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Show `inference.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mjson\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36msubprocess\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36msys\u001b[39;49;00m\n",
      "\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32minput_handler\u001b[39;49;00m(data, context):\n",
      "    transformed_instances = []\n",
      "\n",
      "    \u001b[34mfor\u001b[39;49;00m instance \u001b[35min\u001b[39;49;00m data:\n",
      "        instance_str = instance.decode(\u001b[33m'\u001b[39;49;00m\u001b[33mutf-8\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)        \n",
      "        transformed_instances.append(instance_str)\n",
      "\n",
      "    \u001b[36mprint\u001b[39;49;00m(transformed_instances)\n",
      "    \n",
      "    transformed_data = {\u001b[33m\"\u001b[39;49;00m\u001b[33minstances\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: transformed_instances}\n",
      "    \u001b[36mprint\u001b[39;49;00m(transformed_data)\n",
      "\n",
      "    transformed_data_json = json.dumps(transformed_data)\n",
      "    \u001b[36mprint\u001b[39;49;00m(transformed_data_json)\n",
      "    \n",
      "    \u001b[34mreturn\u001b[39;49;00m transformed_data_json\n",
      "\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32moutput_handler\u001b[39;49;00m(response, context):\n",
      "    response_json = response.json()\n",
      "    \u001b[36mprint\u001b[39;49;00m(\u001b[33m'\u001b[39;49;00m\u001b[33mresponse_json: \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m.format(response_json))\n",
      "    \n",
      "    predicted_classes_str = json.dumps(response_json)\n",
      "\n",
      "    response_content_type = context.accept_header\n",
      "    \n",
      "    \u001b[34mreturn\u001b[39;49;00m predicted_classes_str, response_content_type\n"
     ]
    }
   ],
   "source": [
    "!pygmentize ./model/code/inference.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy the Model\n",
    "This will create a default `EndpointConfig` with a single model.  \n",
    "\n",
    "The next notebook will demonstrate how to perform more advanced `EndpointConfig` strategies to support canary rollouts and A/B testing.\n",
    "\n",
    "_Note:  If not using a US-based region, you may need to adapt the container image to your current region using the following table:_\n",
    "\n",
    "https://docs.aws.amazon.com/deep-learning-containers/latest/devguide/deep-learning-containers-images.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "timestamp = int(time.time())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensorflow-training-201129-2302-005-aec2a92a-tf-1606694522\n"
     ]
    }
   ],
   "source": [
    "recommender_tensorflow_endpoint_name = '{}-{}-{}'.format(recommender_training_job_name, 'tf', timestamp)\n",
    "\n",
    "print(recommender_tensorflow_endpoint_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sagemaker.tensorflow.model import TensorFlowModel\n",
    "\n",
    "tensorflow_model = TensorFlowModel(name=recommender_tensorflow_endpoint_name,\n",
    "                                   model_data='s3://{}/{}/output/model.tar.gz'.format(bucket, recommender_training_job_name),\n",
    "                                   role=role,                \n",
    "                                   framework_version='2.1.0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensorflow_endpoint = tensorflow_model.deploy(endpoint_name=recommender_tensorflow_endpoint_name,\n",
    "                                              initial_instance_count=1, # Should use >=2 for high(er) availability \n",
    "                                              instance_type='ml.m5.4xlarge', # requires enough disk space for tensorflow, transformers, and bert downloads\n",
    "                                              wait=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/endpoints/tensorflow-training-201129-2302-005-aec2a92a-tf-1606694522\">SageMaker REST Endpoint</a></b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "display(HTML('<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/endpoints/{}\">SageMaker REST Endpoint</a></b>'.format(region, recommender_tensorflow_endpoint_name)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# _Wait Until the Endpoint is Deployed_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 190 ms, sys: 15.7 ms, total: 206 ms\n",
      "Wall time: 6min 31s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "waiter = sm.get_waiter('endpoint_in_service')\n",
    "waiter.wait(EndpointName=recommender_tensorflow_endpoint_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# _Wait Until the ^^ Endpoint ^^ is Deployed_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test the Deployed Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from sagemaker.tensorflow.model import TensorFlowPredictor\n",
    "\n",
    "predictor = TensorFlowPredictor(endpoint_name=recommender_tensorflow_endpoint_name,\n",
    "                                sagemaker_session=sess,\n",
    "                                model_name='saved_model',\n",
    "                                model_version=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predict the `star_rating` with Ad Hoc `review_body` Samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'predictions': [{'output_1': [0.0224526152,\n",
      "                               0.0222037733,\n",
      "                               0.0183034446,\n",
      "                               0.0179341771,\n",
      "                               0.017752476,\n",
      "                               0.0173165798,\n",
      "                               0.0167744718,\n",
      "                               0.016768666,\n",
      "                               0.016750684,\n",
      "                               0.0165527295],\n",
      "                  'output_2': ['Half Baked (1998)',\n",
      "                               'Underneath, The (1995)',\n",
      "                               'Baton Rouge (1988)',\n",
      "                               'Leading Man, The (1996)',\n",
      "                               'Firm, The (1993)',\n",
      "                               'Sex, Lies, and Videotape (1989)',\n",
      "                               'Ransom (1996)',\n",
      "                               \"Wooden Man's Bride, The (Wu Kui) (1994)\",\n",
      "                               'Cable Guy, The (1996)',\n",
      "                               'Four Weddings and a Funeral (1994)']}]}\n",
      "['Half Baked (1998)',\n",
      " 'Underneath, The (1995)',\n",
      " 'Baton Rouge (1988)',\n",
      " 'Leading Man, The (1996)',\n",
      " 'Firm, The (1993)',\n",
      " 'Sex, Lies, and Videotape (1989)',\n",
      " 'Ransom (1996)',\n",
      " \"Wooden Man's Bride, The (Wu Kui) (1994)\",\n",
      " 'Cable Guy, The (1996)',\n",
      " 'Four Weddings and a Funeral (1994)']\n"
     ]
    }
   ],
   "source": [
    "from pprint import pprint\n",
    "\n",
    "user_id = \"42\"\n",
    "\n",
    "recommendations = predictor.predict([user_id])\n",
    "\n",
    "pprint(recommendations)\n",
    "\n",
    "pprint(recommendations['predictions'][0]['output_2'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Save for Next Notebook(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stored 'recommender_tensorflow_endpoint_name' (str)\n"
     ]
    }
   ],
   "source": [
    "%store recommender_tensorflow_endpoint_name "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stored variables and their in-db values:\n",
      "blazingtext_test_s3_uri                               -> 's3://sagemaker-us-east-1-835319576252/data/amazon\n",
      "blazingtext_train_s3_uri                              -> 's3://sagemaker-us-east-1-835319576252/data/amazon\n",
      "blazingtext_validation_s3_uri                         -> 's3://sagemaker-us-east-1-835319576252/data/amazon\n",
      "ingest_create_athena_db_passed                        -> True\n",
      "ingest_create_athena_table_parquet_passed             -> True\n",
      "ingest_create_athena_table_tsv_passed                 -> True\n",
      "raw_input_data_s3_uri                                 -> 's3://sagemaker-us-east-1-835319576252/DLAI/amazon\n",
      "recommender_multitask_training_job_name               -> 'tensorflow-training-201129-2249-002-19a0db08'\n",
      "recommender_tensorflow_endpoint_name                  -> 'tensorflow-training-201129-2302-005-aec2a92a-tf-1\n",
      "recommender_training_job_name                         -> 'tensorflow-training-201129-2302-005-aec2a92a'\n",
      "s3_private_path_tsv                                   -> 's3://sagemaker-us-east-1-835319576252/DLAI/amazon\n",
      "s3_public_path_tsv                                    -> 's3://amazon-reviews-pds/tsv'\n",
      "setup_dependencies_passed                             -> True\n",
      "setup_iam_roles_passed                                -> True\n",
      "setup_s3_bucket_passed                                -> True\n"
     ]
    }
   ],
   "source": [
    "%store"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Delete Endpoint\n",
    "To save cost, we should delete the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sm.delete_endpoint(\n",
    "#      EndpointName=tensorflow_endpoint_name\n",
    "# )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
